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Memo 


To: P, McNelis Date : 12-Oct-1988 
From: E. C. May Lpt- Location : G-201 
SG1J Subject : Response to McNelis’ Comments on Analytics . Ce: _————a1 


My response to your comments (18 August 1988) with regard to the then proposed 
addendum to the Enhanced Human Performance Investigations FY 1988 Technical Mid-year 
report (now considered the final report to Objective E, Task 4) will be confined to your specific 


comments 1-13 (see Attachment). 


1. The reference to the FY 1986 is accurate. The work described on page 23 
of the 87 Mid-year Technical Report is a description of work performed 
under Object E, Task 2—Search in FY 1986. (Note page B-9 of the FY 
1987 SOW, Objective F Task 10, “Building upon the results of the FY 1986 
program explore the limits of quality of remote viewing of analytical 
information.”) During FY 1986 there was no specific tasking for the 
analytics program, but I felt that, if successful, it would be applied. in the 
binary search arena, and therefore was a justifiable portion of that task. 


Since “analytics” was not part of the formal tasking, the reporting of it 
appears in the Mid-year Technical Final report for FY 1987 ahead of the 
nd required time of the end of 3rd quarter, FY 1987 (see SOW). 


2. The techniques referred to in (1) and (2) are not necessarily the same. The 
description is confusing as written, so I have clarified that point in the report. 


3&4. As stated, the objectives were confusing, I have reworded them to more 
accurately reflect the SOW and the intent of the project. In my opinion only 
a part of science is of the form of hypothesis formulation, test, and 
reformulation. The creative part of science manifests itself in determining 
the appropriate hypotheses. As part of the FY 1986, SRI initiated this 
concept as part of the search task because we felt it was a good area for 
‘exploration, given the literature and given the application potential, and that 
it met the spirit of Phase I of the contract—knowledge building. In FY-1986, 
we did not send a protocol to the SOC in that to do so within a completely 
unknown area that is being explored would be inefficient. The protocols 
change from moment to moment in that type of environment, and to wait 
until an SOC response is inappropriate in this case I will present this whole 
effort as a specific agenda item during this year’s SOC meeting. The : 
protocols that were used in the formal studies as well as in the exploratory 
studies will be reviewed, and the SOC will determine if the results and 
conclusion presented in the final report for Objective E, Task 4 are justified. 
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5. Yes, this bullet is accurate. I have double checked both our formal reports 
to you with the “raw” data log and find the following. For the three years, 
there has been a total of 2224 binary trials (529—FY 1986, 1168—FY 1987, 
and 527—FY 1988). During that time there were only a total of 382 
non-binary trials (200—FY 1986, 0—FY 1987, and 182—FY 1988). 


6. The differential comparison point has been added to the report in the 
appropriate place. 


7. A minor error was made in the FY 87 Mid-Year. The p-value quoted there 
(0.024) used a normal approximation to the binomial. The value in for final 
report is computed from the exact binomial and is 0.033. The difference is 
of no significance. Thanks for catching this technical error. 


The ideal of sensitivity to putative decline effect was intended to mean that 
the very high baseline (or as rewritten—fiducial point) is not as sensitive to 
increases in hitting rate as it is to decreases, given that a decline effect might 
be true. In any case, this section has been completely rewritten. 


8. As it turned out, you did have valuable input into the final test—auto-typing 
of Ready. This section has been rewritten to reflect the final report nature 
of the document, and all the final trials were completed withing the agreed 
upon deadline. 


9. The wording has been changed to “The hitting rate for these 18 trials was 
78%, p < 1.69 x 107%, and the effect size was 0.50.” 


10. In the final report, all binary trials are displayed, so the confusion in the 
earlier report is no longer present. 


11. The section regarding specific and different internal strategies has been 
expanded for the final report. 


12. What was meant was that since no decline effect was observed, some other 
hypothesis must account for the data. All the conclusions have been 
rewritten to reflect the complete data set. 


13. Along with yours, Jim’s comments have been addressed in the final report. 
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PROJECT 1291 
CLIENT REPORT APPROVAL FORM 
Transcript of McNelis’ Comments Dated August 18, 1988 
(Re. Addendum to FY 1988 Mid-Year Technical Report) 


There are a number of serious concerns I have regarding this document and the actual 


conduct of this experiment which, until clarified and corrected, preclude my giving approval. 


My primary concern is with the lack’of a rigorous design for the work you are doing with 
this subject. In FY 86 he did 50 formal trials on a binary task; follow-up in FY 87 was frustrated 
when, after a large number of pilot efforts, the subject declined formal testing (it is noted that the 
results of the pilot tests were not encouraging). In FY 88 the subject continued to do vaguely 
defined tasks which, in my opinion, will make it impossible to attain the primary objective of 
improving single—-bit hitting rates in binary tasks since (a) base line data are not being collected 
(or, if FY 86 data is being relied on, it is questionable whether 50 trials is sufficient) and (b) we 
will not have the time to run an adequate number of formal trials in FY 88 to demonstrate 


improvement. 
Specific comments follow: 


1. Page 3, first paragraph — reference to FY 86 work in Objective E, Task 2 is 
not accurate. Mention is made to the FY 86 effort for the first time in the FY 
87 Mid-Year Technical Report (page 23) under Objective F, Task 10. 


2. Page 3, Section A, (1) and (2) - are the “techniques” referred to in (1) the 
same as those in (2) or are they external versus internal. Please clarify and 
specify. 


3. Section A, general - multiple objectives are confusing and there is no way one 
could design an experiment at this point regarding the two secondary 
‘objectives since (a) the techniques are not defined, and (b) formal base line 
data has not been gathered. These secondary objectives should have been 
eliminated or more scientifically addressed. 


4. The SOC should have reviewed the “protocol” for this “experiment.” 


5. Page 4, Section C, first bullet - implies a binary tasks is the usual protocol. is 
this accurate since much/most of the data being collected appears to involve 
multiple choices. 


6. Page 4, Section C, last paragraph — alludes to “formal tests.” If single bit 
hitting rates in a binary task are to show improvement, they have to 
demonstrate improvement compared to like base line data and/or include 
sufficient trials over time to demonstrate this. The clarification of this is 
critical. 
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10. 


11. 


12, 


13, 


Page 5, Section D - indicates a p value of < 0.033, while the FY 87 
Mid-Year Tech Report states the p value for the same data is < 0.024. Also, 
this paragraph indicates that the FY 86 data was “sensitive to a putative 
decline effect” - what does this phrase mean and where is the data to support 
it? - 


Page 5, Section E, second paragraph — states “SRI and sponsor’s COTR will 
determine structure of formal test.” This “arrangement” is not in the contract 
and is first mentioned by you in this document’ presented to me in 
mid—August, 1988. I will be happy to input, but see my comment number 4 
above. Also, it may well prove to be too late to do a formal test (see my 
introductory comments and concerns). 


Page 7, Section F, second sentence ~ unclear. Perhaps this should read “The 
task was to determine whether...etc.” 


Page 7, Section F, last sentence - when you say the “rernaining 63 trials” are 
you referring to 63 of the 81 original binary trials? Ifthe first 18 were thrown 
out, then it would seem that your total binary trials are just 63 (especially since 
the first 18 did not fit your “usual protocol” regarding type of targets). 


Page 8, first paragraph, first sentence — what are the “specific and different 
internal strategies” mentioned. This is critical information and, if only Ingo 
knows the answer (assuming he knows), it’s not meaningful to us, 


Page 8, first paragraph, last sentence - what are the “other two hypotheses” 
you are referring to; where are they so stated? 


I have also attached the comments made by QM regarding this 
document. they are also critical of this work and indicate his contention that 
the work is seriously flawed. 
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SG1J 


